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Abstract — Low-cost  sensor  technologies,  such  as  Micro-Aerial 
Vehicles  (MAVs),  present  an  opportunity  to  facilitate  improved 
transformational  Intelligence,  Surveillance,  Reconnaissance, 
Targeting,  and  Information  Operations  (ISRT/IO)  capabilities  in 
support  of  tactical  network-centric  ISRT/IO  Warfighter  and 
First-Responder  operations.  There  is  an  exigency  for  realistic 
rehearsal  frameworks  to  enable  efficient  and  rapid  deployment  of 
MAVs  in  support  of  ISRT/IO;  physics-based  3D  simulation  and 
modeling  capabilities  improve  mission  planning  and  efficiency. 
The  incorporation  of  Free  Open  Source  Software  (FOSS)  such  as 
the  Naval  Postgraduate  School  (NPS)  Autonomous  Unmanned 
Vehicle  (AUV)  workbench  into  the  Sensor-net  Self-Organization 
and  Control  (SenSOC)  initiative  enables  highly  reconfigurable, 
realistic  and  dynamic  mission  planning  capabilities  thereby 
creating  more  responsive  Systems-of-Systems  Engineering 
(SOSE).  The  current  work,  reported  herein,  focuses  on  the 
incorporation  of  a  scalable  sensor-package  in  the  NPS  AUV 
platform-oriented  modeling,  simulation,  and  experimentation 
framework  which  would  support  operational  use-cases  and 
CONOPS  (i.e.  concepts  of  operation)  for  tactical  ISRT/IO 
mission-threads  and  associated  scenarios. 


The  figure  also  illustrates  that  MAVs  enable  man-packable 
sensor-net  capabilities  that,  in  principle,  can  provide  organic 
ISRT/IO  resources  for  individual  stakeholders  in  the  field. 


Figure  1.  Example  Micro-Aerial  Vehicles  (MAVs)  [1  ]-[3] 
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autonomous  vehicle;  sensor-net;  image  processing;  mosaicking; 
georegistration;  modeling  and  simulation;  open  source;  X3D 

I.  Introduction 

Low-cost  sensor  technologies,  such  as  Micro-Aerial 
Vehicles  (MAVs),  present  an  opportunity  to  facilitate  improved 
transformational  Intelligence,  Surveillance,  Reconnaissance, 
Targeting,  and  Information  Operations  (ISRT/IO)  capabilities 
in  support  of  tactical  network-centric  ISRT/IO  Warfighter  and 
First-Responder  operations.  Fig.  1  highlights  only  a  few  of  the 
broad  range  of  MAVs  readily  available  for  tactical  ISRT/IO 
applications  [l]-[3].  Note  that  the  upper  right  image  in  the 
figure  illustrates  remote  launch  and  recovery  of  MAVs  from 
unmanned  ground  vehicles. 


This  tactical  net-centric  ISRT/IO  effort,  reported  herein 
complements,  and  is  working  in  collaboration  with,  robotics 
centers  of  excellence  at  SSC  Pacific  and  elsewhere.  Reviews 
and  surveys  of  related  robotics  work  are  available  [l]-[3].  A 
number  of  standardized  and  interoperable  tactical  robotics 
resources  further  motivate  the  development  of  interoperable 
open-source  mission-driven  sensor-net  framework  capabilities. 
For  example,  Fig.  2  is  a  screen-shot  of  the  Multi-robot 
Operator  Control  Unit  (MOCU)  that  supports  video/imagery 
streams  from  multiple  types  of  platforms  (e.g.  UAV/USV/UGV). 

For  rapid  deployment  and  efficient  utilization  of  such 
emerging  technologies,  especially  in  support  of  tactical 
network-centric  ISRT/IO  Warfighter  and  First-Responder 
operations,  there  is  a  critical  need  for  highly  reconfigurable  and 
dynamic  mission  planning  and  rehearsal  frameworks  with 
highly- integrated  workbench  facilities. 


This  work  has  been  partially  sponsored  by  SSC-Pacific,  as  well  as,  the 
ONR/ASEE  NREIP  Summer  Visiting  Faculty  and  Student  Intern  Program. 


Figure  2.  Screen-Shot  of  MOCU  Configured  as  A  Multi-Vehicle 
(UAV/USV/UGV)  Controller  with  Video  From  A  Manportable  UGV.  [2] 

The  survey  and  initial  results  reported  in  this  paper  are  in 
response  to  a  need  to  utilize  realistic  physics-based  3D 
simulation  and  modeling  capabilities  to  enable  improved 
mission  planning  and  support.  The  inclusion  of  Free  Open 
Source  Software  (FOSS)  such  as  the  Naval  Postgraduate 
School  (NPS)  Autonomous  Unmanned  Vehicle  (AUV) 
workbench  described  in  Section  III  into  the  Sensor-net  Self- 
Organization  and  Control  (SenSOC)  initiative  facilitates  highly 
reconfigurable,  realistic  and  dynamic  mission  planning 
capabilities  thereby  creating  more  responsive  Systems-of- 
Systems  Engineering  (SOSE).  The  additional  incorporation  of 
a  sensor-package  in  the  integrated  FOSS  would  support 
operational  use-cases  (i.e.  concepts  of  operation)  for  tactical 
ISRT/IO  mission-threads  and  associated  scenarios. 

Information  interoperability  (i.e.  sharing)  is  also  an 
inherent,  key  operational  requirement  for  highly  coordinated 
rapid  responses,  as  typically  needed  for  effective  Network 
Centric  Warfare  (NCW)  and  First  Responder  missions  [4].  As 
Fig.  3  illustrates,  improvements  in  information  sharing  directly 
impact  dependent  activities  such  as  shared  situation  awareness, 
collaboration  and  self-synchronization.  Thus,  more 
interoperable  and  seamless  planning,  rehearsal,  and  execution 
capabilities  are  expected  to  dramatically  improve  operational 
mission  effectiveness  for  tactical  ISRT/IO  applications. 


Figure  3.  Tenets  of  Network-Centric  Warfare  (NCW)  [4] 


The  SenSOC  initiative  focuses  on  developing  seamless 
operational  interoperability  and  self- organization  capabilities 
that  include  heterogeneous  ad-hoc  subnets  of  both  human  and 
unmanned  nodes  within  the  networks  of  services  provided  by 
Enterprise  Architecture  (EA)  and  Service  Oriented 
Architecture  (SOA)  frameworks.  In  other  words,  SenSOC 
concentrates  on  the  spectrum  of  Research,  Development, 
Acquisition,  Test,  and  Evaluation/Experimentation  (RDAT&E) 
challenges  associated  with  sensor-net  self-organization  and 
control.  Such  challenges  include  multi-INT  sensor  data  fusion, 
man-packable  sensor-nets,  self-synchronization,  global  MAV- 
based  video  reach-back,  and  most  recently,  semantically- 
enabled  EA/SOA  sensor-net  capabilities  [5]-[15]. 

Fig.  4  is  a  network  diagram  of  a  SenSOC-NPS  experiment 
that  demonstrated  MAV  global  video  reachback  via  the 
Internet  and  commercial  SATCOM  networking  [15].  Thus,  the 
results  highlight  the  potential  extent  and  range  of  networking 
capabilities  that  motivate  the  development  of  extensible  open- 
source  sensor-net  frameworks  that  include  seamless  model 
driven  mission  planning,  rehearsal,  and  experimentation. 


Figure  4.  Network  Diagram  of  Global  Reachback  Experiment  [15] 


In  this  paper,  readily  available  open-source  flight 
simulators  and  3D  modeling  environments  are  investigated. 
The  goal  is  to  utilize  multiple  and  independently  developed 
platform  frameworks  to  support  mission  needs  and  further 
enable  information  interoperability.  A  key  component  that  is 
addressed  includes  a  sensor-package  workbench  for  facilitating 
the  simulation  of  realistic  mission-driven  sensing  capabilities, 
platforms/nodes,  and  communication  networks  that  readily 
support  operational  use-cases  (i.e.  concepts  of  operation)  for 
tactical  ISRT/IO  mission-threads  and  associated  scenarios. 

II.  Needs  and  Emerging  Capabilities 

As  the  previous  section  illustrates,  Warfighter  and  First 
Responder  mission  requirements  necessitate  dynamically 
reconfigurable  sensor  and  communication  networks  that 
provide  reliable  information  for  improved  situational 
awareness  and  rapid  response.  MAV  technology  is  a  high- 
payoff  example  of  emerging  disruptive  technologies  which 
help  further  enable  such  sensor-net  capabilities.  “The  relatively 
low-cost,  easy  deployment,  and  low-detectability  of  MAVs 
provide  viable  next-generation  NCW  and  first-responder 
solutions  for  mission-specific  tactical  ISRT/IO  operations. 


M A V- enabled  sensor-nets  provide  much  needed  high-payoff 
next-generation  ‘eyes  and  ears’  for  Warfighters  and  First 
Responders,  while  keeping  them  out  of  harm’s  way”  [56]. 
Also,  because  of  their  small  size,  MAVs  are  of  great  interest 
for  net-centric  applications,  where,  if  needed,  more  than  one 
MAV  can  be  deployed.  Multiple  MAVs  enable  swarms  and 
other  formations  of  MAVs  to  potentially  self-organize  and 
thus,  more  autonomously  collaborate  while  collecting  and 
communicating  in  the  operational  environment.  UAV  research 
continues  to  be  an  area  of  much  research  and  interest. 
Summaries  of  relevant  examples  of  emerging  inter-dependent 
areas  of  UAV  research  are  found  below. 

Cooperative  MAV  formations  need  to  utilize  global 
information,  efficiently  manage  and  allocate  resources  (MAVs 
and  sensors),  and  be  robust  to  changing  environmental 
conditions  or  mission  requirements  [16].  Multi-UAV  relative 
positions  can  be  estimated  based  upon  each  UAV’s  image  of 
the  scene  and  a  derived  motion  field  of  common  objects  [57]- 
[58].  The  motion  fields  then  enable  the  relative  displacements 
for  each  UAV  to  be  calculated.  Merino  et  al.  utilize  blob 
analysis  to  facilitate  matching  among  the  different  UAV 
captured  images  [57]-[58].  In  [17],  Stolarik,  Niland,  and 
Givens  studied  improving  the  simulated  MAV  communications 
in  the  MultiUAV  [18]  multiple  unmanned  aerial  vehicle  (UAV) 
simulator  developed  by  the  Air  Force  which  can  simulate  8 
UAVs  and  10  targets.  MultiUAV  is  a  MATLAB  based 
program.  Niland  discusses  MultiUAV  being  used  in  the 
FLexible  Analysis  Modeling  and  Exercise  System  (FLAMES) 
to  facilitate  more  complex  Suppression  of  Enemy  Air  Defense 
(SEAD)  planning  [19].  MAVs  which  are  inherently  small  are 
greatly  influenced  by  environmental  disturbances  such  as  wind, 
unlike  the  larger  Predator  for  example.  Research  into  MAV 
dynamics  and  control  should  investigate  wind  effects,  both  on 
control  as  well  as  data  acquisition  [20]-[21].  Yao-hung  and 
Feng  estimate  the  error  in  UAV  position  due  to  the  wind  field 
and  use  this  to  correct  the  UAV’s  position  [20].  Ceccarelli, 
Enright,  Frazzoli,  Rasmussen  and  Schumacher  study  target 
surveillance  by  MAVs  under  wind  disturbances  [21].  Goktogan 
and  Sukkarieh  present  a  hardware  in  the  loop  (HIL) 
environment  to  farther  validate  the  simulated  mission  [22]. 
Sharma  and  Taylor  studied  multiple  UAVs  without  GPS  for 
navigation.  Each  MAV  estimated  its  own  as  well  as  all  other 
MAV  locations  and  orientations.  An  extended  Kalman  filter 
was  used  to  produce  the  position  estimates  [23].  Lin  and 
Goodrich  studied  different  algorithms  for  use  in  search  and 
rescue  path  planning  [24].  MAVs  have  also  been  studied  for 
uses  in  environmental  monitoring  [25]  wildlife  population 
monitoring  [26]  and  wildfire  detection  [27].  Thus,  in  addition 
to  previous  SenSOC  related  efforts,  sensor-package  and  AUV 
workbench  capabilities  are  needed  to  further  enable 
environmental  and  wildlife  monitoring  and  research,  single  and 
group  based  MAV  path  planning  and  coordination,  as  well  as 
disaster  management  [16]-[28]. 

III.  FOSS  AUV  Workbenches  and  Flight  Simulators 
A.  AUV  Workbench 

The  Modeling,  Virtual  Environments  and  Simulation 
(MOVES)  Institute  and  NPS  Center  for  AUV  Research,  Naval 


Postgraduate  School,  Monterey,  California  has  created  an  XML 
based  workbench  called  Autonomous  Unmanned  Vehicle 
(AUV)  workbench  [29]-[35].  This  software  workbench  uses 
Autonomous  Vehicle  Control  Language  (AVCL).  The  AVCL 
includes  definitions  for  pertinent  mission  data  and  parameters 
for  mission  simulations.  AUV  workbench  simulations  for 
vehicle  dynamics  are  determined  utilizing  a  6  degree-of- 
freedom  (DOF)  model.  The  generic  model  can  be  easily 
changed  to  incorporate  new  vehicle  dynamics  or  new  vehicles. 
Background  scenes  during  the  simulation  are  created  using 
Extensible  3D  (X3D).  Missions  can  also  be  mapped  out  using 
the  2D  mission  planner.  Fig.  5  is  a  snapshot  from  the  AUV 
workbench  [29]-[35]. 


Figure  5.  from  MOVES  AUV  Workbench  [29] 


Fig.  6  depicts  a  two  vehicle  3D  mission  planning 
simulation.  This  capability  for  multi-vehicle  simulations  is  an 
advantage  of  AVCL.  AVCL’s  ontology  defines  common 
unmanned  vehicle  function  relationships  that  in  turn  enable 
multiple  vehicle  interactions  in  a  simulation.  This  capability  as 
well  as  the  generic  vehicle  model  facilitates  mission  planning 
with  different  vehicles  [29]-[35]. 


Figure  6.  MOVES  AUV  Workbench  Mission  Planning  Screen-Shot  [29] 

AVCL  defines  the  mission  using  sequentially  executed 
script  commands  or  by  defining  goals  which  are  interpreted  in 
the  form  of  a  finite  state  machine  (FSM).  The  3D  scene 
simulation  is  performed  using  a  scene  access  interface  (SAI) 
This  allows  the  incorporation  of  different  sensor  packages. 
[29]-[35].  Camera  models  however  are  not  included.  This  is  a 
key  component  that  needs  to  be  addressed  so  that  a  sensor 


package  workbench  capability  incorporating  camera  models 
further  enables  realistic  mission  sensor  imagery  and 
communication  data  streams  and  their  associated  problems. 
Bandwidth  limitations  also  play  a  crucial  obstacle  to  full  high 
resolution  scene  imagery  being  transmitted  to  the  AUV’s 
control  station. 

B.  FlightGear  Simulator 

Another  open  source  simulator  is  FlightGear  [36].  The 
simulator  runs  on  many  of  the  commonly  used  operating 
systems  such  as  Windows  and  Linux.  An  important  feature  of 
FlightGear  is  that  it  includes  modifiable  flight  dynamics 
models  as  AUV  workbench  similarly  includes.  Other  options  of 
interest  available  in  FlightGear  include  such  simulator  features 
as  weather,  multi-player,  and  GPS.  FlightGear  is  modified 
easily  by  the  user  by  modifying  XML  files,  similar  to 
modifying  the  AVCL  files  in  AUV  workbench.  This  use  of 
XML  makes  FlightGear  readily  extensible  facilitating 
information  interoperability  across  various  platforms. 

For  initial  proof-of-concept  and  exploratory  assessment, 
FlightGear  possesses  a  surprising  amount  of  interoperability 
due  to  its  utilization  of  XML.  Additionally,  this  exploratory 
assessment  showcased  the  compatibility  of  the  FOSS  simulator 
across  different  operating  systems.  Due  to  availability  and 
more  immediate  ease  of  use,  FlightGear  was  utilized  for 
synthesizing  framesets  used  for  generating  simulated 
mosaicking  results  [3 7] -[3  8],  [56].  A  more  detailed  summary 
and  discussion  of  FlightGear  can  be  found  in  [56]. 

These  initial  results  have  motivated  continued  follow-on 
efforts  to  develop  interoperable  open-source  sensor-net 
frameworks  that  include  extensible  sensor-package  capabilities. 
This  sensor-package  feature  complements  the  platform- 
simulation  capabilities  currently  within  the  NPS  MOVES  AUV 
Workbench  and  other  simulators.  The  following  section 
describes  initial  progress  towards  developing  camera  modeling 
capabilities  as  an  initial  example  sensor-package  component. 

IV.  Camera  model  -  Sensor  Payload 

As  mentioned  earlier,  because  of  their  small  size  and  being 
relatively  inexpensive,  MAVs  are  of  great  interest  for  net- 
centric  tactical  ISRT/IO  applications.  Note  that  potentially,  a 
significant  number  of  MAVs  can  be  deployed  to  create  swarms 
or  other  large  formations  of  MAVs  that  can  rapidly  collect  and 
communicate  data  using  sensor-net  technology  and  smart 
cameras.  Thus,  multiple  MAVs  can  cooperatively  cover  a  large 
area  of  operation  within  a  potentially  much  larger  array  of  other 
fixed  and  mobile  sensors. 

Within  this  context,  an  important  sensor  capability  is  rapid 
collection  and  utilization  of  real  time  image  or  video  scene 
data.  Multiple  scene  images  need  to  be  mosaicked  to  produce  a 
composite  scene  or  aerial  mosaic.  Previous  SenSOC 
experiments  have  demonstrated  that  mosaicking  algorithms  and 
software  developed  for  larger  platforms,  cannot  be  readily 
applied  to  MAV  applications.  These  earlier  results  motivate  the 
development  of  camera  modeling  capabilities  that  help  identify 


and  correct  such  mosaicking  challenges  that  arise  due  to  MAV 
platform  dynamics  and  typical  MAV  camera  characteristics. 
Thus,  initial  information  interoperability  experimentation 
explores  the  feasibility  of  creating  synthesized  video  framesets 
for  more  controlled  analysis  of  video  mosaicking  results.  Initial 
mosaicking  results  from  synthesized  framesets  created  using 
the  FlightGear  simulator  are  found  in  [37]-[38],  [56]. 

Photomosaics  or  image  mosaicking  continues  to  attract  in¬ 
terest  in  the  defense  as  well  as  research  communities  [39]- 
[44].  MAVs  which  are  inherently  small  are  greatly  influenced 
by  environmental  perturbations  and  disturbances  such  as  wind. 
As  a  result,  quality  of  MAV  captured  video  suffers  due  to 
these  disturbances;  this  can  lead  to  issues  in  reliably  and  con¬ 
sistently  mosaicking  MAV  captured  imagery.  To  facilitate 
consistent  image  mosaicking,  Taylor  and  Andersen  geo-re- 
gister  the  images  used  in  the  mosaic  [39].  With  this  geo-regis- 
tration  information,  the  authors’  method  tracks  virtual  points 
between  captured  frames  [39].  Heiner  and  Taylor  also  geo-re- 
gister  the  mosaicked  MAV  captured  images.  However,  the  au¬ 
thors  utilize  information  from  the  MAV’s  Inertial  Measure¬ 
ment  Unit  (IMU)  and  the  Global  Positioning  System  (GPS)  to 
perform  this  geo-registration  [40]  by  use  of  a  common  system 
of  coordinates.  Bulo  and  Birk  demonstrate  the  use  of  an  im¬ 
proved  Fourier-Mellin  Invariant  (iFMI)  for  mosaicking  in 
MAV  as  well  as  underwater  imagery  [43].  A  common  method 
being  employed  for  image  mosaicking  is  the  use  of  the  Scale 
Invariant  Feature  Transform  (SIFT)  to  determine  features 
[41] -[42].  The  SIFT  determined  keypoint  features  are  scale 
and  rotation  invariant.  These  examples  illustrate  some  of  the 
various  techniques  and  methods  employed  in  video  and  image 
mosaicking;  this  demonstrates  the  need  for  a  sensor-net  work¬ 
bench  capability  which  will  enable  and  facilitate  an  improved 
mission  analysis  and  optimization. 

A  camera  model  component  in  the  workbench  will  enable 
realistic  image  acquisition  planning  and  mission  modeling. 
The  camera  model  will  need  to  incorporate  various  parameters 
of  interest  such  as  operational  theater  and  mission  specific  val¬ 
ues,  camera  system  physical  and  electrical  characteristics,  and 
environmental  conditions.  To  address  this  need,  perspective 
effects  on  the  image  have  been  examined  for  a  generic  camera 
model.  Perspective  induced  effects  “affect  the  size  and  direc¬ 
tion  of  patterns  to  be  recognized”  [45]  and  so  must  be  accoun¬ 
ted  for  [45]-[46].  Fig.  7  shows  the  effect  of  camera  tilt  along 
the  X-Z  plane  on  terrain  area  represented  by  each  pixel  [38]. 
The  relative  distance  represented  by  each  pixel  is  plotted  in 
Figs.  8  and  9  [38].  To  ensure  correct  image  mosaicking  or  tar¬ 
get  identification,  identification  of  objects  or  processing  of 
pixels  near  image  edges  will  need  to  account  for  this  nonlinear 
distance  representation  as  well  as  possible  distortion  due  to  the 
camera  optics.  Fig.  10  shows  the  percent  overlap  between  suc¬ 
cessive  frames  as  vehicle  speed  changes  on  a  straight  path  at 
constant  altitude  with  the  imager  held  parallel  to  terrain  [38]. 
These  preliminary  camera  model  results  highlight  the  added 
value  for  including  a  sensor-package  workbench  capability  in 
the  FOSS  or  workbench. 


Figure  8.  (Left)  Camera  imaging  system  parallel  to  the  surface  [38]. 
(Right)  Relative  distance  represented  by  each  pixel 


Figure  9.  (Left)  Flat  Camera  imaging  system  at  angle  to  the  surface  [38]. 
(Right)  Relative  distance  represented  by  each  pixel. 


Figure  10.  Percent  overlap  between  successive  frames  as  vehicle  speed 
changes  on  a  straight  path  at  constant  altitude  (imager  parallel  to  terrain)  [38]. 


V.  Summary  and  Conclusions 

As  discussed  and  highlighted  in  the  previous  sections,  there 
is  a  critical  need  for  highly  reconfigurable  and  dynamic 
mission  planning  and  rehearsal  frameworks  that  are 
interoperable  and  support  extensible  sensor-workbench 
facilities.  Such  FOSS  frameworks  enable  rapid  deployment  and 
efficient  utilization  of  emerging  disruptive  sensor-net 
technologies.  In  particular,  frameworks  with  sensor-packages 
that  include  camera  modeling  features,  provide  operationally 
realistic  test  beds  that  can  be  utilized  for  realistic  MAV  based 
image  and  sensor  systems  simulation  and  modeling 
development  in  support  of  tactical  network- centric  ISRT/IO 
Warfighter  and  First-Responder  operations. 

VI.  Future  Work 

In  addition  to  further  exploring  FOSS  video  and  image 
processing  resources,  such  as  OSSIM  and  other  FOSS 
image/video  processing  resources  [47]- [48],  semantic  features 
of  the  MOVES  Military  Scenario  Definition  Language, 
(MSDL),  Coalition  Battle  Management  Language  (CBM-L) 
and  other  recently  developed  capabilities  [49] -[5 3]  are  also  of 
particular  interest  for  integration  with  a  companion  SenSOC 
effort  called  Enterprise  Lexicon  Services  (ELS)  [8] -[9]. 
Semantic  capabilities,  such  as  MSDL,  CBM-L,  and  others 
identified  within  the  context  of  ELS,  are  of  great  interest  due  to 
potential  improvements  in  information  interoperability  that,  in 
turn,  enable  provenance  and  pedigree  capabilities  [54]  for 
E  A/SO  A  based  sensor-nets  and  related  services.  Finally,  a 
long-term  goal  includes  incorporating  multi-objective 
optimization  tools  [55]  into  interoperable  sensor-net 
frameworks  for  best  matching  mission  needs  and  resourcing 
constraints  with  sensor-packages,  platform  features,  and  the 
growing  wealth  of  FOSS  and  commercial  COTS/GOTS  sensor- 
net  related  technologies. 
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